DOCUMENT RESUME 



ED 459 766 



IR 058 260 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



AVAILABLE FROM 
PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Cadenas, Paula 

Evaluation of Multimedia: Why and How? An Expedition in 
Three Voyages . 

2001-08-00 

8p.; In: Libraries and Librarians: Making a Difference in 
the Knowledge Age. Council and General Conference: 

Conference Programme and Proceedings (67th, Boston, MA, 
August 16-25, 2001); see IR 058 199. 

For full text: http://www.ifla.org. 

Reports - Descriptive (141) -- Speeches/Meeting Papers (150) 

MF01/PC01 Plus Postage. 

Electronic Publishing; *Evaluation Criteria; Foreign 
Countries; *Instructional Material Evaluation; *Multimedia 
Materials 

Digital Data; Venezuela 



ABSTRACT 



This paper narrates the three years of experience of the 
Banco del Libro ' s (Venezuela) Committee of Evaluation of Digital Formats. It 
describes how this institution, after 40 years of experimentation with the 
printed word, has made its move into the digital domain. The parts of digital 
formats are summarized, including: the medium; authorship data; technical 
data; and synopsis, general content, and function for informative and 
recreational publications. Criteria for the evaluation of interactive 
multimedia resources are presented, including: typography, color, 
illustrations and pictures; animation; multimedia; and interactivity. Some 
new reflections about reading and other findings related to digital formats 



are also discussed. (MES) 
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Abstract: 
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The pages that follow intend to narrate the 3 years of experience of the Banco del Libro ’s Committee of 
Evaluation of Digital Formats. It is the story of how this Venezuelan institution , after 40 years of 
experimentation with the printed word, has dared to make its move into the digital domain,. By means of 
the chronicle of three voyages, we describe in detail the criteria for multimedia evaluation that we have 
devised, some new reflections about reading and other findings from digital lands. 



Plut 6 t que de condamner les jeux de video, les humanistes, les pedagogues, les 
createurs, les auteurs devraient s ’emparenter de cette nouvelle ecriture et produire avec 
elle des oeuvres dignes de ce nom, inventer les nouvelles formes de savoir et d’ exploration 

que lui correspondent, lui donner ses lettres de noblesse. 

Pierre Levy 1 



The Banco del Libro is a Venezuelan institution whose foundations are laid on an apparently 
simple premise: reading is pleasure. But it also represents forty years of voyage which have never ceased 
to be fruitful, despite the ups and downs of trial and error. It has confided in an open and innovative 
exploration, instead of adopting uni vocal and infallible truths, since they would go against the very 



1 “Instead of condemning video games, the humanists, the pedagogues, the authors, the creators, should become 
familiarized with this new sort of writing, produce with it works worthy of that name, invent its corresponding new 
forms of knowledge and exploration, and give it its title of nobility.” 
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concept of reading for which it stands. For the institution - and particularly for the groups of study, 
evaluation and promotion of books for children and youths - reading is the hazardous act of interpreting 
and building sense from the individual. But to see such romantic idea followed up by promotion projects 
in our Latin American reality has been in fact what has nurtured us. For reading is a complex inter- 
individual practice (Petit, 1999) and at the same time a social one (Ferreiro, 1995). 

It is precisely in this innovative spirit that the project of Cd-Rom evaluation is conceived. 
Because we started off with the certainty that reading, not only in the written text or in the images of a 
picture book, but also in a conversation or in front of the screen of a computer, is an unstoppable process. 
Studying these new forms of reading, and selecting the best of the CD’s which already accompany the boy 
from a tender age, are part of the work of the current evaluation group. And how do we know - especially 
today, in the middle of this revolution of concepts and ideas - what can be deemed as good or bad? We do 
not pretend to issue magic formulas, nor easily applicable prescriptions, but to form criteria; we do not 
pretend to impose our selection, but to provide guidance. Next I will try to narrate to you, by means of a 
chronicle, the voyage endeavored by the Banco del Libro: the departure, the evaluation criteria that have 
guided the course, along with some reflections on the itinerary and, finally, where has our voyage led us 
so far. 



Planning the departure ... 

It was three years ago that the Committee of Evaluation of books for children and young put 
forward the idea of expanding its activities to the area of electronic publications. This idea arose from the 
fact that children’s books and multimedia had many points in common. 

First, an investigation of the multimedia market was carried out, and contacts were established 
with publishing houses that could be interested in collaborating with this young project. They responded 
promptly. And we decided to begin with the reviewing of stories in Cd-Rom, following the premise that 
the kind of reading to which the electronic book invited was closely linked to that of the recreational book. 

We also found out that, during the eighties, along with the fruitful publication of educational 
software, numerous study and evaluation groups had arisen, conformed in their majority by teachers, 
parents and librarians. And it called our attention that they were in their majority study groups sharing the 
same views: multidisciplinary approach was not the rule. 

We wondered then about the possibility of gathering specialists in different disciplines, and 
proceeded to do exactly that. Once the variegated crew was assembled, the journey was planned, and the 
day of meeting was fixed, we departed in our first exploratory voyage. But, as we advanced in the 
discussions, and carried out the process of nurturing the Banco del Libro’s Center of Documentation with 
theoretical material and a collection of multimedia titles, the initial course of the project was modified. 

The first turn of the trip, and perhaps the most significant one, was marked by the realization that the thirty 
years of experience of the Committee of Evaluation of books and its criteria were not directly translatable 
to this domain. Evaluating books for children was not equivalent to evaluating electronic stories. 
Contradictorily, the lack of clarity of these beginnings caused the enrichment of the team. Because only 
deepening in the revision of many CD's, and thanks to a plural and multidisciplinary approach, we could 
finally comprehend the complexities of their nature. Those were days of rejecting prejudices, of tracing 
limits between the reading of a book and those experiences to which the digital world could be inviting us. 
The unavoidable fear to the threat of the new gradually vanished, while curiosity invited us to the revision 
of apparently “solid” traditional ideas. And we started to open up to words arising with the fast pace 
inherent to the reality they meant to designate: hypertext, interactivity, virtuality and globality, zapping, 
interface and rupture of linearity... The idea was - and it still is - to grant to this language of the new world 
a space for the caution and the depth; before it is drained in utopias and generalities. 
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Then we weighed anchors for the second time, but with a more open itinerary: the book and the 
CD-Rom invited to different communicational relationships. We understood that the wealth lies in the 
eco-systemic possibility (Calderon, 1999), in the “MacLuhanian” sense of “works-bridges” in continuous 
feedback, which expand the reader's communicational experience: a book leads to the movie, and from 
there to the multimedia and eventually to the web page, or vice versa (Banco del Libro, 2000a). One 
medium doesn't have to annul the experience offered by another, quite on the contrary, the formal 
execution of each work from its own language opens the interpretation possibilities and the aesthetic 
enjoyment. Between the years 1999 and 2000, we dared to carefully revise all the titles that fell in our 
hands, not discarding publications in other languages, because they contributed to give us a richer 
panorama of the medium’s status. We were able to create new fields in the records and to systematize the 
entry of the CD's in the databases, establishing also an evaluation procedure which included report and 
review, in case it was recommended. Study lines and biannual timetables were devised to establish 
classifications and types of CD's, trying to understand the internal rules of these new genres. The 
restructuring of the Banco del Libro’s web site was supported and oriented. And a group of CD's was 
rewarded in the Banco del Libro 2000’s event “The Best Books for Children and Young”, being 
eventually used in activities and projects of reading promotion with new media. 

The crew had returned after its second voyage with some discoveries and achievements: a set of 
criteria that we continually validate until today, categories and classifications of the evaluated products, 
the design of records with new entry fields in the databases and the publication of reviews, essays, articles 
and reports narrating the experiences and different visions of this initial journey. Already at the end of last 
year (Banco del Libro, 2000b) approximately 45% of the CD's we revised were recommended for their 
quality, of which 20% were in other languages. 35% of the studied publications were discarded, and 20% 
were considered of improvable quality. In these cases, we have tried to establish contact with the 
publishers - especially Venezuelans and Latin Americans - in order to send them the report, and we have 
even managed to hold nutritious exchange interviews with them. Many other titles exceeded our area of 
study, but they were nevertheless revised. In the second stage, we went through a careful pre-selection, in 
order to be able to study the panorama better. For this reason, out of 3 revised titles, 2 were recommended. 
Today, we have enlarged the focus again, and we revise publications for children and young, not just 
educational, but of all type. Curiosity, open-mindedness and flexibility, instead of rigidity and dogma, 
have guided the course of the evaluations. 

So far, the group of evaluation of digital formats continues sailing in its third voyage, with no 
return date. This crew, more adventurous and expert than in the beginning, continues its in-depth 
exploration of the new digital world. And it is today, after three years of experience, when we feel ready 
to share with you the outline of criteria that has guided us in the course (Banco del Libro, 2000a; Ladron 
de Guevara, in press). 

Criteria for evaluation of digital formats: a map to analyze the parts without losing sight of the 
whole. 

The medium 

To start the evaluation, it is important to get acquainted with the physical medium that contains 
the information (CD-Rom, DVD, web site, video game). In view of the changing nature of these media, it 
is convenient to analyze their advantages and disadvantages, in order to offer recommendations for their 
use and to compare them with the different platforms in which the same information is presented. The 
evaluation could eventually focus on the relationship between different products created under the same 
title, for instance: Cd-Rom Lego Creator and web site with activities (Banco del Libro, 2000a). 

Authorship data: 

In electronic publications, the different areas and disciplines that intervene in their creation should 
be taken into account. To the traditional authorship centered on the text and illustrations, now we should 
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add the authorship of the design of navigation, interactivity and animation, digital videos, music, special 
effects and programming (Banco del Libro, 2000a). Thus, as in a movie, the spectator can recognize 
behind a well achieved scene the hidden stamp of one or several creators. 

Our experience has led us to verify that quality is directly linked to the coherent and articulate 
work of a team of professionals of different disciplines. Perhaps this is how certain products manage to 
stand out of the snub marketing formulas that eventually collapse in the anonymity. Naively, we tend to 
believe that the multimedia has abolished the author. But the idea of authorship is perhaps being 
reconsidered. Would multimedia produce then the multiplicity of authors? Although the new relationships 
that could be arising in the author-reader symbiosis are currently part of our reflections, they represent an 
entire line of investigation to which we should grant a space apart. Let us get on the way again... 

Technical data 

They refer to the technical requirements necessary for the execution of the electronic 
publication. This information is usually in the packing of the product itself, but it does not necessarily 
guarantee a clear and specific recommendation. It is in this area that the evaluator has demonstrated to 
play a primordial role, because his or her reading should go beyond the seduction offered by the packing, 
and should also determine with more precision and detail the true technical demands, besides giving 
‘hints, tips and tricks’ and other computer trifles that should be included in the review (Banco del Libro, 
2000a). 

Synopsis, general content and function 

Here we intend to seize the product’s intention through the synthesis of its contents: the 
argument or central thematic axis. After tracing a summary of what it seeks to communicate - of its script 
-, we should be able to proceed to examine with more thoroughness its chosen language and strategies. We 
intend to become aware of the subject and the genre (Ladron de Guevara, 2000). For example, it is not the 
same thing to write or read poetry or narrative; the quality of an atlas is not judged as that of a novel, 
because they have different contents and internal rules. It helps, in principle, to use the enlightening - but 
sneaky - classification of publications in informative and recreational: 

Informative 

The publications classified as informative usually include the consultation and educational genres. 

a. Among the publications of general consultation, there are the dictionaries and interactive 

encyclopedias; the popular virtual visits and the lists of references would also qualify. 



b. The educational products are usually classified as tutorials, developers of capabilities and 

explorative. 



Recreational 

The recreational publications can be grouped in at least three types: related with literary genres, 
workshops and games. 

a. Under the category of titles related with literary genres we usually have the electronic 

stories, living books, movie books and more classical works; derived from theater plays or poetry, for 
example. 
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b. The workshops, on the other hand, furnish the user with scenarios and tools meant to 

propitiate creation. 



c. Finally, the games are applications especially designed to entertain, whose ludic end 

remains even when they have a plot based on, say, history. Here we usually include imaginative electronic 
versions of table games, “challenge chains” and strategy games. 

Evaluation of interactive multimedia resources 

In this level of analysis, it is specially important to determine how each resource works 
separately, and then to judge its relevancy, function, articulation and definitive result regarding the whole. 
Due to the popularity of these products, the stereotype becomes frequent. And with the novel sound, 
animation and interactivity tools, the market is saturated with fiascoes. They usually don't go beyond a 
supposedly attractive appearance, their content is innocuous, the resources are underemployed and the 
interactivity promise is not fulfilled. But these are products that end up being drained in themselves thanks 
to the wisdom of natural selection. 

Typography, color, illustrations and pictures: 

Here the graphic and diagramming aspects are reviewed, as well as the quality of digital 
resolution. Author's illustrations that enrich the plot, pictures of real facts supplementing a CD on history, 
could be subtle aesthetic choices helping to enhance the final quality of the publication and the persistence 
of the user’s enjoyment. 

Animation: 

This area concentrates on analyzing the dynamic nature of the images: animation of icons, series 
or sequences, flying logos and anything that implies movement as a tool to strengthen the 
communicational strategy. It is a resource that can contribute to the definition of the characters in the 
stories. For example, highlighting a certain way of walking or the repetitive and peculiar movement of a 
character’s hand can convey psychological depth and humor to the narrative structure. Again, it would be 
a matter of assessing the relevance of the elements introduced in the screen. Because gratuitous 
animations only overcrowd memory, and are usually limited to the amusing intention, without any 
relevance regarding the general structure. 

Multimedia: 

Here the incorporation of sound objects is analyzed: incidental music - environmental and/or 
descriptive -, voices and dialogues, off-screen narration, sound effects accompanying the interaction, 
digital sound videos. In this field, the quality of the music, its originality and references can positively 
influence the quality of the publication. However, the musical background of each screen or activity is 
usually neglected, often resorting to the same hollow and repetitive tune. There is a perceivable difference 
when sound, image and movement come together to expand the initial intention of the script, to 
supplement the possibilities of sense. 

Interactivity: 

Here it is convenient to go over the classical notions of activity and passivity (Ladron de Guevara, 
in press). In principle, the interaction with a work refers back to the activity to which it invites. And it can 
be appreciated in a book, a painting or a sculpture. But the multimedia comes to offer, as in a movie, a 
third dimension. The program is supposed to determine the user's freedom of exploration, and to take into 
account complex variables such as the time and the ideal speaker's psychology. This goes without a doubt 
beyond the classical connotations of the author-reader relationship, and leads us to the author's implicit 
dialogue with his imaginary reader, or perhaps as well to a patent verification of the semiotic relationship 
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of the text with its human target. Evaluating, for example, the experience to which an educational software 
invites us could be equivalent to evaluating the interaction between the schoolboy and the teacher 
(Buckleitner, 1 999). 

In this sense, the selected mechanisms are reviewed to unveil the information and the navigation 
strategies outlined in the publication, in order to assess how they are employed to expand the experience. 
Analyzing the hypertext can be fundamental, for example, for the evaluation of an encyclopedia, because 
the universe of knowledge would be enriched with the user’s possibility of interlacing connections and 
elaborating, in the individual exploration, his own corpus of sense. The interactivity levels can define the 
open or closed character of the publication. Then the possibility arises of analyzing the interactivity levels 
as well, some centered on the exploration, and others, for instance, on the user's creative will. It is true that 
the interactivity offer can be seen as a continuum, a long line: in one end, the closed universe planned by 
the programmer, in the other one, the “freedom” each explorer is able to reach. (Banco del Libro, 2000a; 
Ladron de Guevara, in press). 

Communicational experience 

In this port, we integrate again the fragments of analysis. We assemble the parts in order to 
perceive the whole. We started out with the intention expressed by the publishing team, and now we 
recapitulate to see the final result in the ‘how’. Art critics usually remind us that the most important thing 
is the how, not the what, because in our time the same subjects have been discussed over and over again. 
The innovation is in the treatment given to the execution and its ulterior effect. Here the author-reader - or 
the multimedia set-explorer - dialogue is reestablished. It is finally in the reception, in the interpretation 
where, more or less, you can measure the quality. 

This section has to do with the final conclusion and summarizes the general impression of the 
evaluator (or group of them): how is the function of an electronic publication related to the formal 
aspects? Which strategies allow themselves to be seen? Has it been possible to transmit what we suppose 
was intended? (Banco del Libro, 2000a). 

It is necessary to consider the possible balance between the intention of the publication and the 
media selected to achieve it. Here it can be accurately mentioned how the electronic publication expands 
the literary experience with the incorporation of animation, music and interactivity options. We can also 
study the nature of this experience and wonder about more extra-textual aspects: to what codes it appeals, 
how much complicity it requests to the user, how much novelty it proposes and, very especially, how it 
dialogues with the books, the cinema, the oral narration or the storyteller, or how it can supplement other 
school or family activities. 

The turn ... 

It is in fact with the communicational and aesthetic experience, with the turn to the reader, to the 
subjectivity, that we close the evaluation. Our starting point was the individual experience with the work, 
then we examined the parts in objectivity trials, we figured out sense in the analysis, and we returned to 
ourselves to elucidate what the encounter left us. After having displayed our map of criteria, let us wisely 
remember that the individual experience is absolutely essential. It is, for example, in the reviews of books 
and recommended CD's where we intend to be most transparent about the process that we go through: 
each evaluator writes down his or her experience and view of the work. These lines allow to see the subtle 
agreement between shared criteria and individual tastes. Couldn’t this be a definition of reading? Is 
perhaps evaluation another name for profound and analytical reading? Is it not the invitation to make a 
critical selection a way of promoting reading in its own right? 

The evaluation program invites to probe beyond appearance, trying to figure out senses in the 
intimate dialogue between the reader and the work. The support of the institution consists of propitiating a 
space of continuous formation for each evaluator, for each reader, putting him or her in contact with a 




6 



7 



diversity of styles and manifestations, creators, critics and publishers. This year, for instance, we have 
been able to systematize even more the concept of formation, because, parallel to the weekly evaluation of 
the two committees, we have programmed seminars, workshops and conversations with authors and 
publishers, while each evaluator keeps weaving his or her reflections in articles and reviews for both 
Banco del Libro’s and external publications. 

We finish with an invitation to come closer to us, and to participate in Caleidoscopio , the Banco 
del Libro’s digital magazine, in http://www.bancodellibro.org.ve. There we intend to offer - from the core 
of the semantic wealth of the word ‘reading’ - a space for the exchange, the debate and the formation. 

Forty years of experience and study have revealed to us that it is an absurdity to speak of reading if we 
banish individuality. For reading is in the man. 
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